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EXECUTIVE SUMMARY 


The Free Library of Philadelphia's Literacy Enrichment Afterschool Program (LEAP) is a key part of 
Philadelphia's comprehensive citywide Out-of-School Time (OST) system. For more than 25 years, the 
Free Library has offered children academic support and enrichment through LEAP, a free, voluntary, 
and drop-in program. At the request of the Free Library of Philadelphia, Policy Studies Associates 
(PSA) conducted an evaluation of LEAP during the 2018-19 program year. 


Focused on the experiences of LEAP participants in PreK-3, the evaluation explored a central 
hypothesis: Drop-in programming in a literacy-rich environment can help young people develop the 
skills they need to succeed in school and beyond. Drawing on data collected through interviews with 
LEAP staff and librarians, program observations, surveys of students and of staff, and analysis of 
participation records, the evaluation explored the contribution of LEAP to the citywide OST system 
and the benefits of LEAP participation. Key findings and our recommendations for further 
strengthening the program are summarized below. 


LEAP offers a distinct contribution to the Philadelphia OST System 


A drop-in program accessible to all 


LEAP is grounded in the Free Library's mission to “advance literacy, guide learning, and inspire 
curiosity” and in the public library commitment to be an open and accessible community resource. 
Reflecting these values, LEAP is committed to its role as a drop-in program, available to all children 
without expectation for daily attendance. LEAP encourages children and families to engage with the 
program based on their needs. While some children attend LEAP daily, other families do not need or 
want to commit to a daily program but benefit nonetheless from the resources and assistance 
available on occasion. For families like these, LEAP offers important flexibility and complements other 
structured OST opportunities available in Philadelphia. 


Indeed, participation records suggest that LEAP welcomes both a core group of regular participants 
and occasional participants: of the 1,538 PreK-3 children recorded as coming to a LEAP program in 
2018-19, 27 percent attended for more than 20 days of programming, and 25 percent attended only 
once. This pattern of utilization is reflective of LEAP’s goals and expectations, and we recommend 
that metrics for LEAP’s success and contribution to the Philadelphia's OST system reflect the value of 
serving families and children in a range of ways; assumptions underlying typical OST participation 
measures are inconsistent with LEAP design. 


An opportunity for whole-family learning 


Located in public libraries, LEAP sites have a unique opportunity to create an inclusive learning 
environment that meets the needs of both children and families, especially since Free Library requires 
that an adult accompany young children (7 and under) to LEAP. In our observations, LEAP staff 
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inconsistently leveraged this opportunity to create a space where a family can learn together: in 
some libraries, adult caregivers play only the role of a chaperone while LEAP staff work with the child. 
However, in some programs we saw LEAP staff leverage the presence of these caregivers to amplify 
the impact of the program. For example, LEAP improved the capacity of parents to support their 
child's learning by modeling homework assistance strategies, directing them to additional resources, 
and connecting them to a wide array of community resources. 


A literacy-rich environment 


LEAP offers a literacy-rich environment. Although LEAP is not a learn-to-read program, the library 
setting connects young people to books, and staff encourage positive reading behaviors and self- 
confidence. Children can access resources and computers to complete school projects and find 
books that match their interests and reading levels. Literacy is often embedded in program activities 
involving homework assistance, in which students receive individualized support, to arts or STEM 
activities, which often are based on a book or incorporate a storytelling component. Although these 
literacy-embedded strategies are evident in many LEAP sites, additional support and training from 
the Free Library can further build the capacity of LEAP staff to consistently and intentionally 
implement high-quality literacy supports. Many of the embedded literacy activities that we observed 
were locally developed by LEAP staff and librarians. 


LEAP helps participants develop literacy competencies 


LEAP is not a learn-to-read or a literacy tutoring program; there is no single “LEAP model” and 
specific activities vary from library to library. However, each day LEAP offers space for children to 
complete their homework, receiving individualized assistance as needed. LEAP staff also plan a daily 
project that provides a mix of academic support and enrichment. Through these supports, LEAP aims 
to provide a safe and supportive environment, help children learn to access library resources, and 
help children develop a love of reading and gain self-confidence and self-reliance for learning. 


A safe and supportive environment for learning 


Libraries serve as safe havens in many communities. LEAP strives to create a welcoming environment, 
foster supportive relationships for young people with staff and peers, and offer both physical and 
emotional safety to participants. LEAP staff promote a youth-driven program through flexible 
structures that allow students to engage in planned activities or to guide their own program 
experience. LEAP staff reported both helping children process challenging situations in their lives that 
can impede learning, and also offering children individualized encouragement for learning and 
reading in a judgement-free zone. 


Increased self-confidence for learning 


In surveys, LEAP participants in grades PreK-3 generally reported high levels of self-confidence for 
learning: they responded positively to statements like “When | try, | can get good grades in school” 
and “At LEAP, | learn how to be a good learner.” LEAP sites can foster this confidence by creating a 
positive culture around literacy and learning. We also observed LEAP staff engaging children in 
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enjoyable activities linked to reading and providing individual encouragement and assistance in 
practicing their skills, whether through homework help or other activities. 


Improved access to knowledge and resources to support learning. 


LEAP is positioned to offer children access to and use of library resources, including computers, to 
support their learning and schoolwork. We observed LEAP staff supporting this skill-building by 
working with children on school projects; for instance, using library resources to conduct research or 
helping them access books, encyclopedias, and biographies for their assignments. 


LEAP staff would benefit from additional literacy training. 


We observed evidence of supportive environments and promising strategies across the LEAP sites 
visited for the evaluation; survey data from all sites also suggest that across libraries, PreK-3 
participants have positive experiences in LEAP and benefit from their participation. 


However, many of the promising practices that we observed were locally developed by LEAP staff 
and librarians, who also reported that they were eager for additional training and resources on 
supporting literacy. We recommend that the Free Library offer additional training and resources— 
including peer-learning and mentoring for LEAP staff—to institutionalize and spread intentional 
literacy strategies across LEAP sites. 


The Free Library of Philadelphia’s Youth Services and Programs already manages LEAP, hires and 
trains LEAP staff, and provides resources to LEAP sites. With additional guidance, the Free Library can 
support the scaling of these strategies across programs, deepening the impact of LEAP on early 
literacy: 


m Leveraging the role of library as a community institution by encouraging the involvement of 
caregivers and parents and modeling effective strategies for supporting children’s learning. 


# Being intentional about and reinforcing literacy connections and learning goals, even when 
literacy is embedded into enrichment activities. 


m Drawing on trusting relationships and knowledge of children’s interests to encourage 
reading. 
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INTRODUCTION 


On a typical afternoon, children and youth of all ages go to neighborhood libraries throughout 
Philadelphia to receive homework assistance, participate in activities that enhance literacy and STEM, 
and spend time in a safe, resource-rich community space. Some come with friends or siblings, some 
with their parents, and some by themselves. The Free Library of Philadelphia has offered young 
people academic support and enrichment throughout the City of Philadelphia for more than 25 years 
through a free, voluntary, drop-in program: the Literacy Enrichment Afterschool Program (LEAP). 


LEAP staff—part-time Afterschool Leaders (ASLs), LEAP Maker Mentors (LMMs), and Teen Leadership 
Assistants (TLAs)— receive monthly curriculum training and guidance on topics or activities that can 
be adapted to reflect the neighborhood context, local cultures, and grade levels served by nearby 
schools. For younger children, LEAP staff offer a story break or read-aloud during homework time; 
some libraries engage participants in makerspace activities. LEAP also gives young people access to 
library computers and draws on partners to offer special activities or events. 


In 2018, at the request of the Free Library of Philadelphia, Policy Studies Associates (PSA) launched 
an evaluation of LEAP. Our evaluation was designed to advance understanding of how LEAP supports 
literacy-rich environments for children in Pre-Kindergarten through the third grade (PreK-3). We 
examined and assessed the implementation and impact of strategies intended to support LEAP’s 
learning outcomes. This report summarizes the findings from the evaluation and offers 
recommendations to further strengthen LEAP’s effectiveness and impact in Philadelphia. 
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BACKGROUND ON LEAP 


LEAP is one component of Philadelphia’s comprehensive Citywide Out-of-School Time (OST) system, 
designed to promote positive development and outcomes for youth and improve the quality of life in 
Philadelphia's neighborhoods. 


Philadelphia's OST System 


According to the city's 2018-2026 strategic plan, Philadelphia invests more than $24 million annually in 
OST programs through a collaboration of the City Managing Director's Office, the Department of Human 
Services, Parks & Recreation, the Free Library, the School District of Philadelphia, and other private and 
public organizations that together provide a range of opportunities before and after school and during 
the summer. 


Though its OST initiative, Philadelphia seeks to build a high-quality system of services and supports that is 
“measured, coordinated, collaborative, and embraces continuous improvement models in order to 
provide youth with tools they need to be successful Philadelphians.” As part of these quality-improvement 
efforts, for example, the City has worked with the National Institute for Out-of-School Time (NIOST) to 
identify strategies that OST providers can use to support improved student literacy outcomes, and is 
developing a framework for quality improvement in the Philadelphia OST system. A 2012 grant from The 
Wallace Foundation supported the development of a data management system for the City’s afterschool 
providers, including the Free Library of Philadelphia's LEAP program. The strategic plan also promotes the 
alignment of OST with other city initiatives, including the Read by 4" literacy campaign, highlighting the 
goal that all Philadelphia children will read on grade level by the time they enter the fourth grade. 


OST in the Public Library Context 


The mission statement of the American Library Association (ALA) emphasizes its commitment “to 
enhance learning and ensure access to information for all.” In pursuit of that mission, the ALA has 
embraced core values of access and of confidentiality/privacy (ALA, 2006). Public libraries are 
committed to ensuring that “all information resources that are provided directly or indirectly by the 
library, regardless of technology, format, or methods of delivery, should be readily, equally, and 
equitably accessible to all library users.” Libraries are also committed to protecting the confidentiality 
and privacy of all users. 


Grounded in these values, the Free Library of Philadelphia offers a unique contribution to 
Philadelphia’s OST system. The Free Library has intentionally made LEAP available to all young people 
in grades PreK-12, and there is no expectation for daily attendance; this drop-in design distinguishes 
LEAP from most other City-funded OST programs. 
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How does the Literacy Enrichment Afterschool Program (LEAP) contribute to 
the City of Philadelphia’s Out-of-School Time (OST) system? 


The City of Phila delphia set a vision: Philadelphia’s Out of School Time system delivers high-quality 
programming to all low-income and at-risk children and youth, with an emphasis on K through 4, to promote their 
individual safety and wellbeing as well as teaching them the skills they need to be successful in school and in life. The 
OST system engages the community to create a more civically engaged Philadelphia. 


Philadelphia invests in and provides policy guidance to OST programs, including LEAP. 


2018-26 strategic plan created a citywide vision and mission 
Structure for coordinating the city’s programs 

Supports for quality-improvement 

Data management system 

Coordination with broad city initiatives 


The Free Library of Philadelphia provides a resource-rich environment for LEAP. 


The Central Library provides: 

* Management of LEAP, the largest program operated through Youth Services and Programs 

* Training to library staff, Afterschool Leaders (ASLs), Associate Leaders, and Teen Leadership Assistants (TLAs) 
* Curricula resources and guidance on LEAP themes 


Regional and branch libraries offer: 

* Safe, welcoming environment for participants and families 
* Support and engagement from all library staff 

* Access to books, electronic resources, and technology 

* Programming tailored to community needs 


LEAP staff, including ASLs and TLAs bring: 
¢ Near-peer mentoring 
* Passion and spark 


LEAP is a free, drop-in program that offers access to OST programming 5 days a week at the 54 regional and branch 
libraries in Philadelphia. In 2017, youth in grades K-12 participated in LEAP for a total of more than 112,000 days. 


Near-peer mentoring 
Participant-directed learning 
Literacy enrichment 

STEM and Maker programming 
Homework assistance 

Career development 


LEAP participants and Teen Leadership Assistants engage in safe, supportive environments and 
develop literacy skills* foundational to success: 


Knowledge about library resources 
Collaboration and teamwork skills 
Self-confidence and self-reliance for learning 
Strong relationships 


Through their involvement in LEAP, TLA’s also increase: 
* College and career awareness 
* College enrollment 


*Literacy is the ability to identify, understand, interpret, create, compute and communicate using visual, audible and digital 
materials across disciplines and in any context. (from International Literacy Association, www. literacyworldwide.org) 
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With the adoption of the Philadelphia's OST data management system, the Free Library encourages 
LEAP staff to collect enrollment forms, including demographic data, and to track daily participation. 
However—aligned with its commitment to open and free access to all—the Free Library does not 
require families to complete enrollment forms before participation. In addition—reflecting the library's 
value of privacy and confidentiality—many LEAP staff are reluctant to require young people to sign in 
or to track their participation and personal information in the data management system. 


As aresult, standard measures of success of OST programs, including benchmark goals for enrollment 
and attendance, are inconsistent with the vision and implementation for library-based, drop-in 
programs such as LEAP. 


LEAP Staffing and Design 


The Free Library's Youth Services Division operates LEAP and offers materials, guidance, and training. 
LEAP staff are hired and trained centrally by the Free Library's Youth Services Division and benefit 
from the resources available through the central library and through local librarians. 


In 2018-19, 51 of the 54 libraries in Philadelphia hosted a LEAP program. At each library site, LEAP is 
directed by a part-time Afterschool Leader (ASL) or LEAP Maker Mentor (LMM) who manages day- 
to-day programming, supported by two or three Teen Leadership Assistants (TLAs), high school 
students hired for between four and 10 hours per week. These LEAP staff work under the guidance 
and supervision of the librarian, although the level of involvement of the librarian varies by site. 


Importantly, there is no single “LEAP model.” Specific activities vary from library to library, although 
all LEAP afterschool programs are grounded in the Free Library's mission to “advance literacy, guide 
learning, and inspire curiosity” and its vision “to build an enlightened community devoted to lifelong 
learning.” 


Each day LEAP offers space for children to complete their homework, and LEAP staff provide 
individualized assistance as needed. LEAP staff also plan a daily project that provides a mix of 
academic support and enrichment. LEAP emphasizes a youth-driven program environment, with 
LEAP staff designing program activities based on the needs and interests of the local community. 
This approach is consistent with research showing that high-quality OST learning environments offer 
a blend of academic and developmental skill-building activities by using approaches that allow for 
youth autonomy and choice (National AfterSchool Association, 2014). 


Anticipated Impacts of LEAP 


By design, LEAP does not deliver targeted and intensive literacy instruction. Rather, as described 
above, it provides literacy and learning supports in an informal, caring environment through the use 
of engaging enrichment activities and individualized assistance with schoolwork. 
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LEAP also places high value on social and emotional learning (SEL). In recent years, SEL has garnered 
increased attention as an important foundation in students’ development and learning. The Every 
Student Succeeds Act (ESSA) has supported SEL by granting states flexibility in how they define 
student success and emphasizing safe, healthy, and supportive environments (CASEL, 2019). OST 
settings such as LEAP are flexible, holistic, and conducive to building relationships Jones, Bailey, 
Brush, & Kahn, 2017). Evidence suggests that youth benefit from participation in SEL-focused 
programs; a meta-analysis found that youth had improvements in their perceptions, attitudes, 
behaviors, and academic performance (Durlak, Weissberg, & Pachan, 2010). Programs focused on SEL 
skills have also been shown to be effective in improving students’ sense of connection to school and 
achievement (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011). 


In addition, a review for the U.S. Department of Education concluded that positive learning 
environments can help students develop grit, tenacity, perseverance, and other traits that lead to 
success (Schechtman, DeBarger, Dornsife, Rosier, & Yarnall, 2013). That review highlighted three 
competencies in particular that can ensue from positive learning environments: (1) students’ ability 
to perceive themselves as successful learners; (2) effortful control, in which students learn to focus on 
the task at hand, and (3) actionable skills for students to deal with challenges. 


Other researchers have found that those competencies can contribute to learning success. In a 2012 
framework, the University of Chicago Consortium on Chicago School Research identified academic 
mindsets, social skills, academic perseverance, learning strategies, and academic behaviors as key 
factors supporting student success (Farrington et al., 2012). 


The Free Library expects that children who participate in LEAP will benefit from experiences, 
mindsets, and skills that are foundational to literacy and learning success: 


m Safe and supportive environment 

m Positive interactions with peers and adults 
m Love of reading 

m™ Knowledge of library and literacy resources 


m Self-confidence and self-reliance for learning 
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OVERVIEW OF THE LEAP 
EVALUATION 


PSA's evaluation of LEAP explored a central hypothesis: Drop-in programming in a literacy-rich 
environment can help young people develop the skills they need to succeed in school and beyond. The 
evaluation focused on the experiences of participants in grades PreK-3 and addressed five main 
questions about several aspects of the implementation and impact of LEAP: (1) program utilization 
and operations; (2) strategies to promote foundational literacy and social-emotional skills; (3) 
leveraging of the Free Library's resources; (4) effects of LEAP on the development of youth's skills 
and attitudes; and (5) recommendations for improvement. 


PSA used the following methods to gather data and information for the evaluation: 


m Administered an online staff survey in February 2019 to LEAP staff (ASLs and LMMs) and to 
librarians at all Free Library branches with active LEAP programs. 


m Surveyed LEAP participants over a 10-week period between February and April 2019. The 
survey used age-appropriate features such as a “smiley face Likert scale” and yes/no 
questions, and was administered in the form of “Question of the Week” cards. 


m Analyzed data from the Philadelphia Citywide OST database, including participant 
characteristics and records of LEAP program attendance. 


™ Conducted one-day site visits to 10 LEAP sites in February and March 2019, including 
interviews with LEAP staff and librarians and observations of LEAP programming. 


The specific research questions and more detailed explanation of methods can be found in Appendix A. 
The remaining sections of this report discuss evaluation findings about LEAP implementation and 
LEAP’s strategies and benefits, followed by conclusions and recommendations. 
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FINDINGS ABOUT LEAP 
IMPLEMENTATION 


Many factors affect the implementation of OST programs. For LEAP, factors internal to individual 
libraries can impact implementation, including the experience and roles of staff and the size of and 
resources available in the library. External factors, such as the OST needs and priorities of either the 
community in general or a specific group of children who might attend on any given day can also 
affect the types of activities that staff can implement. The drop-in nature of LEAP amplifies these 
factors as staff anticipate and plan for fluctuating patterns of program engagement and utilization 
and adapt to the local needs and resources of their library site. These factors in turn influence how 
staff deliver LEAP programming, and how intensively participants engage with that programming. 


In this section of the report, we present findings about the staff leading LEAP programs and about 
the patterns of LEAP utilization, drawn from staff survey data and from Philadelphia‘s OST 
participation data system. 


staffing 


Both LEAP staff and librarians contribute to implementation. 


LEAP staff—ASLs and LMMs—are charged with day-to-day implementation, with guidance from the 
Free Library's Youth Services Division. Librarians provide additional oversight and help LEAP staff and 
participants leverage library resources. In surveys, LEAP staff and librarians confirmed these different 
responsibilities within LEAP programs, although their roles sometimes overlap. 


All LEAP staff reported that they were responsible for: (1) designing and planning LEAP 
programming, (2) leading LEAP activities, and (3) offering homework help. A majority also reported 
that they play a role in supervising others, such as TLAs (81 percent) and providing resources for 
LEAP activities (77 percent). 


Librarians reported being primarily responsible for supervising LEAP staff (61 percent) and 
connecting the LEAP program to resources (55 percent). 


Some librarians also reported supporting LEAP participants directly. Children’s librarians were more 
likely to report this involvement (45 percent) than were other librarians (30 percent). The most 
commonly identified forms of direct involvement with participants included assisting with LEAP 
activities and offering homework help. However, librarians do not typically play a core role in the 
design or delivery of LEAP activities. Of the 75 librarians responding to the survey, only eight 
reported that they design or plan LEAP activities and six reported leading LEAP activities. 
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The limited experience of many LEAP staff points to the 
importance of training and support. 


Reflecting the high turnover common to OST programs and the part-time nature of working in LEAP, 
38 percent of LEAP staff reported that they had started in their role at the beginning of or during the 
current program year; just 6 percent had 10 or more years of LEAP experience (Exhibit 1). 


To acertain extent, the involvement of librarians can help to provide stability and continuity of 
quality to the LEAP program. Forty-four percent of librarians had more than 10 years’ experience with 
LEAP, either in their current library or another library; an additional 12 percent of librarians had six to 
10 years of program experience. 


Exhibit 1: Number of years of experience with any LEAP program, 
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Exhibit reads: In 2018-19, 38 percent of LEAP staff were in their first year in their role with the program. 


Given the central role that LEAP staff play in the design and delivery of LEAP programs and the more 
limited support role of librarians, there is a clear need for professional development. The Free Library 
offers monthly training opportunities on curriculum and biannual professional development 
meetings for LEAP staff. 


However, LEAP staff need support and training to build their capacity, particularly if a goal of LEAP is 
to support the literacy development of young children and contribute to Philadelphia's Read by 4‘? 
goal. In survey responses, 48 percent of LEAP staff and 53 percent of librarians said that too little or 
far too little training was provided in early literacy (Exhibit 2). 
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Exhibit 2: Perceptions of available resources 
to support high-quality LEAP programming, in percentages 


Training for staff in literacy instruction 


LEAP Staff (N=40) 50 2 
Librarian (N=68) 47 
} } HE Far too little 
Staff to help design and deliver literacy-related activities HEE 100 little 
LEAP Staff (N=46) 70 4 | About right 
Librarian (N=70) 58 Too much 
t 
Literacy-focused materials and resources 
LEAP Staff (N=48) 73 4 
Librarian (N=67) 70 | 
! 
0 20 40 60 80 100 


Exhibit reads: Forty-eight percent of LEAP staff report that there is either “too little” or “far too little” training for staff 
in literacy instruction, as do 53 percent of librarians. 


LEAP staff that we interviewed consistently identified early literacy as an area in which additional 
support and guidance is needed, as most LEAP staff do not have a background in education or 
training in children’s literacy. As summarized by the experience of one ASL: “When | am helping a 
child read, | am going on my memory of learning phonics. It would be helpful to have a refresh on 
that throughout the City. | don’t know a lot about that instruction. | have picked it up intuitively.” 


Participation and Youth Engagement with LEAP 


With the advent of the Philadelphia OST system and the citywide OST data management system, 
LEAP staff have been encouraged to track the characteristics and daily attendance patterns of 
individual LEAP participants, as have their counterparts from other OST programs in the City. 
However, this data entry is inconsistent across libraries, perhaps reflecting both the library value of 
confidentiality and the limited time that the part-time LEAP staff can devote to data entry. 


Indeed, as discussed above, LEAP does not require families to complete enrollment materials for 
their child to participate as is typical in most OST programs. Many LEAP staff do collect demographic 
and participation data—particularly for those who attend regularly and with whom they develop 
trusting relationships—but these data are not complete. Overall, approximately three-quarters of 
LEAP participants have demographic data entered by LEAP staff in the data management system. The 
proportion of participant records with missing race/ethnicity ranges from 0 percent in 10 libraries to 
69 percent in one library. Therefore, while the analyses presented below provide some insight into 
the characteristics of LEAP participants and their patterns of utilization, they should be interpreted 
with caution, and likely undercount participation. 
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As a drop-in program, LEAP welcomes both a core group of 
participants and occasional participants. 


According to data captured in the City of Philadelphia's OST data system, 4,465 young people 
participated in LEAP in the 2018-19 program year, including 1,538 children in PreK-3 (39 percent) 
(Exhibit 3). 


For many of these children, LEAP is a regular 
afterschool activity: 419 PreK-3 participants 
(27 percent) attended at least 20 days in 
2018-19, and 110 attended 75 or more days 
(7 percent). Others come to LEAP on occasion, 
when they need assistance or for a special 
event: 73 percent of PreK-3 children 


Exhibit 3: LEAP participants, by grade, 
2018-19 


EJ PreK-3rd 
( 


n=1,538) 


participated in LEAP for fewer than 20 days, 4th-5th 
including 25 percent who attended just once. (n=1042) 
By design, LEAP welcomes both the core Sih -Bth 
group of regular participants and occasional (n=1,074) 
participants. 

os is 9th-12th 
Participants and families also take advantage (n=268) 


of the supports offered by LEAP over multiple 

years. Of the 1,538 LEAP participants in K-3 in 

ane) oF ea perceny had also attended Exhibit reads: Thirty-nine percent of LEAP participants in 
LEAP previously, including 112 who were 2018-19 were in PreK-3 (1,538 participants). 

three-year participants. This pattern of 

attending LEAP over multiple years was evident both for regular participants and occasional 
participants: the number of days attended in 2018-19 ranged from 1 to 156 for PreK-3 participants 
who had attended in at least one previous year.' 


LEAP serves a diverse population of learners in Philadelphia. 


Available data indicate that LEAP serves a racially diverse population, including for the PreK-3 
participants who are the focus of this evaluation: of those for whom data were entered, 40 percent 
were identified as Black/African American, 22 percent as Asian, and 15 percent as Latino/Hispanic 
(Exhibit 2). The data also point to opportunities for LEAP to increase its outreach to certain 
communities in Philadelphia: a comparison to the demographics of the PreK-3 population in the 
School District of Philadelphia, from which LEAP draws the majority of its participants, suggests that 
LEAP serves a notably lower proportion of Black/African American and Latino/Hispanic children than 
are enrolled in the district (Exhibit 4). 


1 Our data are limited to the 2015-16 to 2018-19 program years. It is possible that some participants first attended 
LEAP prior to the 2015-16 program year. 
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Exhibit 4: Race/ethnicity for LEAP participants 
and School District of Philadelphia, 2018-19 


All LEAP PreK-3 LEAP PreK-3 in the School 
Participants Participants District of Philadelphia 
(N=3,391) (N=1,140) (N=42,997)2 

Asian 15.8% 21.7% 7.9% 
Black/African American 49.7 40.4 47.1 
Latino/Hispanic ARIES 14.6 23.0 
White 10.7 13.3 15.6 
Multiple races 11.9 9),7/ 6.1 

Other race 0.3 0.4 0.3 


Exhibit reads: In 2018-19, approximately 16 percent of LEAP participants were identified as Asian in the Philadelphia 
OST data management system. 


Safety and homework help are primary reasons for coming to 
LEAP. 


In survey results, PSA looked for reasons that might help to explain youth participation—and 
persistence—in LEAP. When asked to report on reasons for youths’ initial attendance at LEAP, 83 
percent of staff ranked “Youth need help with homework" as one of the top three reasons, and 79 
percent ranked “Youth need a safe place to go after school.” 


While these reasons continued to be important in their rankings of why youth choose to become 
regular participants, LEAP staff also reported that participants value the relationships fostered with 
other participants and with staff, as well as the enrichment activities that LEAP offers. Fifty-two 
percent of staff ranked “Youth develop a relationship with or connection to LEAP staff” as one of the 
top three reasons for regular participation, and 51 percent identified “Youth develop friendships with 
other youth” as a driver of regular participation in LEAP (Exhibit 5). 


* Source: School District of Philadelphia Demographics and Enrollment, 2018-19 school year. 
https://www.philasd.org/performance/programsservices/open-data/school-information/#district_enrollment 
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Exhibit 5: Staff reports of participants’ reasons for initially attending LEAP 
programming and for becoming regular LEAP participants 


Reasons i _ attend LEAP (N=120) Reasons youth become bias LEAP participants (N=116) 


83% Youth need hide with homework 70% 


0 61% 


Youth need help completing a special project 


Youth are relatives of or friends with 
other participants 


Youth develop a relationship with or 
connection to LEAP staff 


n/a | Youth develop friendships with other youth 
Other family members are — 
attending an activity at the library 


4% Youth enjoy the chance to read 
° the selection of books at the library 


100 


80 60 40 20 0 0 20 40 60 80 100 


Exhibit reads: Eighty-three percent of LEAP staff ranked “Youth need help with homework" as one of the top three 
reasons youth initially attend LEAP programming. Seventy percent of LEAP staff believe that homework help is also a 
primary reason for becoming a regular LEAP participant. 


Note: Staff were not asked to rank “Youth develop friendships with or connection to LEAP staff” or “Youth develop 
friendships with other youth” as an initial reason for participation. Similarly, they were not asked to rank either “Youth 
are relatives of or friends with other participants” or “Other family members are attending an activity at the library” as 
a reason for youth becoming regular LEAP participants. 


PreK-3 participants themselves identified similar reasons for attending LEAP: more than 90 percent 
reported attending because LEAP offers a safe space, because they like the staff, and because LEAP is 
fun. In addition, 83 percent come to LEAP for help with schoolwork. Children were less likely to 
identify reading or specific LEAP projects as a motivation for attending (Exhibit 6). (To ensure that 
analyses of the “Question of the Week” survey data best capture the experience of regular LEAP 
participants—who were best positioned to report on their program experience—we excluded 
responses from participants who checked “This is my first time at LEAP” on their response card.) 
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Exhibit 6: PreK-3 participant reports of reasons for attending LEAP 


It's a safe place (n=280) 

LEAP is fun (n=280) 

| like LEAP staff (n=280) 

For help with school work (n=281) 

To be with friends (n=281) 

Read books or listen to stories (n=280) 

For a special activity or Maker project (n=280) 
| came with family (n=278) 


Use the computer and the internet (n=277) 


ee) 
Xs 


j=) 


96% 


93% 


93% 


8 


80% 


77% 


76% 


75% 


63% 


10 20 30 40 50 60 70 80 90 100 


Exhibit reads: Ninety-six percent of PreK-3 LEAP participants responded that one of the reasons that they 


attend LEAP is because “It’s a safe place.” 
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FINDINGS ABOUT STRATEGIES 
AND BENEFITS OF LEAP 


LEAP serves young people whose families and neighborhoods are experiencing rapid change, with 
increasing poverty in some neighborhoods and gentrification in others (Bowen-Gaddy, 2018; Pew 
Charitable Trusts, 2019). Philadelphia also reflects the demographic shifts underway nationally. In 
2017, 13.4 percent of Philadelphia residents were born outside of the United States, including 5 
percent of school-age children. Nearly a quarter (22.5 percent) of school-aged children in 
Philadelphia speak a language other than English at home (U.S. Census Burau, 2017). In the School 
District of Philadelphia, Hispanic enrollment has grown by 4 percent in the last 10 years (School 
District of Philadelphia, 2019). This increasing diversity requires strategies to respond to the diverse 
learning, linguistic, and social-emotional needs of all youth. 


LEAP is well-positioned to support the youth of Philadelphia, including those from under-resourced 
communities. LEAP provides academic support and opportunities to develop relationships with 
caring adults. LEAP can connect families with resources in their communities—such as counseling 
and mental health services—to better support their children (Valladares & Ramos, 2011). LEAP can 
also help youth who are English learners practice their language skills in an informal learning 
environment (Holstead & Doll, 2015). To do so effectively, LEAP staff strive to develop relationships 
with the communities served and to understand what families want and need to support their 
children’s cognitive, intellectual, and social-emotional development. 


LEAP aims to create a strong foundation for literacy and learning by offering a safe and enrichment 
environment, with supportive and collaborative interactions with adults and peers. Though LEAP, the 
Free Library promotes increased knowledge of how to access and use library resources, and 
increased self-confidence and self-reliance for learning. Our analyses explored the ways in which 
LEAP achieved these goals for PreK-3 participants, and the strategies that support these benefits. For 
these analyses, we drew on reports from the staff survey and interviews, as well as on responses to 
the participant Questions of the Week.? 


3 Our analyses focus on the responses of PreK-3 participants, excluding those who checked that they were attending 
LEAP for first time on the Question of the Week card completed. We compiled responses for each question to 
determine the average response on key domains. We also aggregated these participant responses to determine the 
average rating on each domain within each library. 
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Creating a Foundation for Learning 


LEAP provides a safe and supportive environment for learning. 


Libraries often serve as safe havens for communities, and the LEAP program pays particular attention 
to providing welcoming and inclusive environments for the children and families served. 


Across LEAP sites, children in grades PreK-3 reported high levels of physical and emotional safety. 
Participants gave an average rating of 2.91 on a scale of 1-3 to the statement “| feel safe at this 
library”; the average rating across the three questions asked about safety and sense of belonging 
was 2.82 on a scale of 1-3 (Exhibit 7). 


Exhibit 7: PreK-3 participant reports of LEAP as safe and supportive 


Average rating across 
all questions (N=940) 2.82 


| feel safe at this library (N=309) 2.91 
LEAP is a comfortable place 284 
to hang out (N=321) : 
| feel like | belong here (N=310) 2.65 
1 2 3 


Exhibit reads: PreK-3 LEAP participants gave “I feel safe at this library” an 
average rating of 2.91 on a 3-point scale. 


The perception of LEAP as safe and supportive was positive across LEAP libraries: the average within- 
library rating for this domain ranged from 2.42 to 3 out of 3, with an average within-library rating of 
2.85. 


Research shows the value of creating a supportive learning environment to promote success. Programs 
such as LEAP can help children deal with the effects of adverse and traumatic experiences, such as 
economic hardship; parental divorce, incarceration, or death; or exposure to violence, drugs and 
alcohol, or mental illness (Sacks & Murphey, 2018); these adverse experiences can have long-lasting 
consequences on health, well-being, and success (SAMHSA, 2013;SAMHSA, 2014). Indeed, the effects 
of childhood trauma are observed in education settings where youth have learning difficulties (i.e., 
poor concentration, memory, organization, and language) and behavior problems (O'Grady, 2017) 
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In a community library located in a neighborhood where poverty has increased, according to the 
librarian, LEAP provides a needed safe environment for children during afterschool hours. Parents 
and caregivers are “so appreciative of the program and thankful that [children] have a place to go for 
help with school,” the librarian noted. “Kids know that we are here to help with homework. 
Otherwise, for some, the library is a nice, safe, warm, and cozy space to be.” LEAP programs also 
foster community connections. An ASL from a library in a rapidly gentrifying community noted that 
as a free program hosted in a trusted community institution, LEAP offers “a unique opportunity to 
bring together kids of different backgrounds.” 


Helping Young People Be Ready to Learn 


In Lower North Philadelphia, LEAP staff described a surrounding community dealing with issues of 
violence and poverty while also slowly gentrifying. The children’s librarian lauded the value of LEAP in 
helping young people process these challenges: 


There have been shootings nearby or close enough that kids have seen them 
when they come in from school. I've had students come in talking with me or 
[LEAP staff] who have a family member who’s been shot. It's processing that 
trauma for them and with them, which is a lot of emotional support to take on, 
but it’s very real and it happens all the time. 


The program strives to establish a “neutral” environment that is removed from the violence children may 
experience or witness in the community; one staff member characterized the site as a “Switzerland” 
where participants can feel free to be themselves and be safe. In addition, staff establish expectations for 
behavior and model and reinforce positive peer interactions. They defuse bullying and other negative 
behaviors by acknowledging that while young people may feel “this need to stick up for yourself or 
defend your friends,” at LEAP “you don't need to,” one staff member said. “We've got you. We'll help 
you.” In keeping with this approach, staff reported that when a behavior issue arises, rather than kicking 
participants out of the LEAP program, they talk through what participants are feeling and experiencing 
and give them space and time to adjust their attitudes and behaviors. 


Staff at this site emphasized that their program was “youth-driven,” meaning that staff are flexible with 
activity plans if participants are “not feeling it” or express interest in something else. This approach is 
consistent with research demonstrating that OST programs can help youth overcome adverse 
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Fostering an Inclusive Learning Environment for Children and Families 


In a program located in a diverse, multilingual South Philadelphia community with a mix of long-term 
residents and recent immigrants, LEAP staff emphasized the importance of creating an inclusive learning 
environment that meets the needs of both children and their families. Indeed, research suggests that 
educating parents and connecting them with resources and services in an environment where they are 
respected and heard can help counter the challenges faced by many youth and families (SAMHSA, 2013). 


A cornerstone for this program is working with both parents and children, helping parents support their 
children’s learning by modeling support strategies, directing them to resources, and providing other types 
of valuable assistance, while also being mindful that English is not the primary language for many families 
served. For example, on the day of our visit, a parent asked the ASL to explain a school permission form 
before they signed. 


According to the ASL: 


The parents are also there to work with the kids. Creating that learning community and 
knowledge community ts really important when talking about homework assistance and 
navigating the school system. 


LEAP participants have positive interactions with peers and staff. 


A recent review of evidence of the effectiveness and characteristics of early literacy OST programs 
identified a safe emotional climate that fosters positive relationships as a key condition for success 
(Hartmann, Comly, Reumann-Moore, & Bowditch, 2017). LEAP participants gave high ratings on 
questions about their interactions with peers and with staff, with an average rating of 2.75 out of 3 
(Exhibit 8). 


Exhibit 8: PreK-3 participant reports of positive interactions at LEAP 
Average rating across 
all questions (N=889) 2.75 
LEAP staff care about 
me (N=327) ate 
At LEAP, | like working 
with the TLAs (N=235) etB 
| have a lot of friends 
at LEAP (N=327) 2.70 


Exhibit reads: PreK-3 LEAP participants gave “LEAP staff care about me” an average rating of 
2.82 on a 3-point scale. 
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By library, the average participant rating ranged from 2.33 to 3 on the 3-point scale, with a mean 
within-library rating of 2.77, suggesting these positive interactions were common across libraries. In 
our interviews, we learned about strategies that LEAP staff use to intentionally promote these 
relationships, consistent with research confirming that effective OST programs have an explicit 
emphasis on the development of staff-youth relationships Jordan, Parker, Donnelly, & Rudo, 2009; 
Moore, Bronte-Tinkew, & Collins, 2010). 


In some cases, these positive relationships were strengthened by staff becoming familiar with the 
social fabric of the library community. For example, one ASL commented that many in the 
community know each other and are extended family—either through kinship as “distant cousins” or 
through quasi-family relationships, such as “play cousins.” In another site, the librarian described how 
staff members make a point of actively greeting all community members as they arrive and knowing 
them by name; this librarian added that having TLAs who live in the neighborhood also helps to 
increase the sense of connectedness. 


LEAP staff are actively engaged with children on a personal level, instilling a sense of trust and 
bolstering the child’s connection to the program. In one program, we observed staff publicly praising 
participants for their work; for example, LEAP staff calling to the librarian, “There’s some really good 
reading going on over here” and inviting the librarian to come listen to the child reading. 


Forging a LEAP Community 


“The number one thing is to make [a new participant] feel as welcome and 
included as someone who has been here 30 days. How are you, where do you go 
to school, what grade are you in? It’s trying to be friendly and generous and 
giving. Help a kid become comfortable in learning in a new space.” —ASL 


Creating community is not happenstance. This ASL explained: “| do my best to have positive personal 
interaction with every youth participant and accompanying adult, every day they attend. | demonstrate 
interest and curiosity in their lives just as they do mine. And | make it my main purpose to make sure that 
LEAP day-to-day meets their expressed needs, [and] not only my agenda.” 


In our observations at this site, we saw LEAP staff personally greeting students as they came into the 
library throughout the afternoon. They engaged with participants on a personal level, asking about cousins 
and new baby brothers. 


LEAP staff also checked on participants’ need for support as they settled in to do their homework—asking 
questions like: “What are we working on today? Or “Do you need help, or are you OK on your own?” Staff 
sat and worked individually with students in a relaxed atmosphere, on tasks ranging from word searches to 
readina aloud. 


LEAP staff design daily enrichment activities that offer occasions for participants to engage ina 
common project; for PreK-3 participants, this typically includes sharing materials or playing a game, 
rather than creating a joint product. Nearly all—92 percent—of LEAP staff planned activities or 
games for students to build vocabulary, and 50 percent regularly planned for children to read out 
loud to one another. 
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These types of activities allow children to work respectfully and collaboratively with others on 
building foundational literacy skills; across all LEAP sites, PreK-3 participants gave an average rating 
of 2.77 out of 3 to questions about their ability to work well with others (Exhibit 9), with average 
ratings by library ranging from 2.36 to 3 (mean of 2.77). LEAP staff and librarians also observed the 
benefits of working together at LEAP: nearly all—94 percent—reported improvements in the ability 
of LEAP participants to communicate with others and to express ideas verbally. 


Exhibit 9: PreK-3 participant reports of working together 


Average rating across 
all questions (N=1,828) 2.77 


| treat other people 
with respect (N=312) 2.85 


| work well with others in 


completing an activity (N=342) 277 


When working together, 
| listen to what everyone Qht 
thinks (N=304) 


| try to give helpful 
suggestions at LEAP (N=238) 2.13 


| learn more when | work 


with other students (N=327) 2.72 
When | work in a group, my 272 
work habits improve (N=305 " 
J 
1 2 3 


Exhibit reads: PreK-3 LEAP participants gave “I treat other people with respect 
an average rating of 2.85 on a 3-point scale. 


” 
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Developing Literacy Competencies 


Within this supportive environment, LEAP aims to help participants develop literacy competencies, 
including a love of reading, knowledge of library resources, and self-confidence. Although LEAP, by 
design, is not a learn-to-read or literacy tutoring program, literacy is integrated into activities ranging 
from homework help to arts. As one LEAP staff member put it, ASLs “try to connect literacy to 
everything we do—whether it is a current event article, a comic, an autobiography activity.” 


Participants respond positively to LEAP activities designed to 
encourage reading. 


All LEAP sites embed literacy skill-building into homework help, according to staff we surveyed. 
Nearly all reported embedding reading (93 percent) or writing (92 percent) into other activities and 
encouraging silent or independent reading during program time (93 percent). In addition, 87 percent 
offer literacy-related computer games and 81 percent provide story time for LEAP participants. 


One ASL summed up his approach as “sneaky literacy,” meaning routinely sneaking literacy into fun 
and engaging program activities rather than into an explicit focus. “We do a lot of storytelling and 
narrative thinking, without necessarily opening a book and saying, ‘You have to read along with me 
and do 20 minutes of reading,”” said this ASL. “We are trying to introduce reflective thinking and 
narrative thinking even into the art that kids are making. Kids that are building with Legos and 
blocks, it turns into a story.” 


PreK-3 participants gave “LEAP makes reading fun" an average rating of 2.77 on a 3-point scale 
(Exhibit 10). By library, the average rating ranged from 2 to 3 (mean of 2.66), indicating some 
variation in the success of LEAP staff to implement engaging literacy activities. 


Exhibit 10: PreK-3 participant reports of love of reading 
cae ie: a .. 
LEAP makes reading fun (N=378) fs 277 EIGN 
| like reading (N=321) | sd 2.63 : CR 
| like to write (N=309) fs 2.49 “ : ; / ‘ i ‘ i ‘ - \y 


1 J 
1 2 3 


Exhibit reads: PreK-3 LEAP participants gave “LEAP makes reading fun 
“an average rating of 2.77 on a 3-point scale. 
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Intentionally Integrating Literacy in LEAP 


In LEAP sites where participants rated their love of reading most highly, staff used various strategies to 
promote reading. Across these sites, acommon thread was intentionally planning for literacy. For 
example: 


# Regularly reading with children to provide individual attention and encouragement to young 
readers: “We read with almost every child. When they like books, when they like reading, they 
usually have good results at school.” 


H Presenting books in appealing ways, such as through monthly, theme-based book displays to 
spark interest: “A lot of times that’s how [a regular participant] picks his books: he just walks 
around and sees what's on top, and what's interesting to him.” 


@ Individualized literacy skill-building, such as working with a participant to write a poem 
about seasons; collaborating to come up with rhyming words helps to build language and 
literacy skills. 


@ Embedding reading in arts or STEM activities, like creating “space slime” after reading a 
book for Women’s History Month about Mae Jemison, the first African-American woman to go 
into space, or creating squares for a Freedom Quilt after reading books honoring Black History 
Month. 


# Leveraging partnerships, as one LEAP site did for an activity in which children selected a book 
to read to a therapy dog that visited the library. 


LEAP helps children build skills to access and use library resources. 


LEAP staff help children learn how to access and use library resources to support their learning and 
schoolwork. Nearly all LEAP staff and librarians reported helping participants with computer searches 
for research (97 percent) and choosing books matched to their interests and reading levels (99 
percent). A large majority of staff reported that these types of activities have helped to improve 
participants’ digital knowledge (82 percent) and have led to students checking out more library 
books (69 percent). 


PreK-3 participants across LEAP sites reported that coming to LEAP helped them with schoolwork 
(average rating of 2.8 out of 3), although they were somewhat less likely to report knowing how to 
use the computer to look up information, perhaps reflecting their young age (Exhibit 11). The 
average rating for this domain by LEAP program ranged from 2 to 3 on a scale of 1 to 3 (mean rating 
across libraries of 2.66). In highly rated sites, we observed the librarian helping students access 
books, encyclopedias, and biographies to assist students with specific assignments. 
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Exhibit 11: PreK-3 participant reports of knowledge of library resources 


Average rating across 266 
all questions (N=647) , 


Coming to LEAP helps me 2.80 
with my school work (N=305) 


| know how to use the computer 252 
to look up information (N=342) , 


1 2 


Exhibit reads: PreK-3 LEAP participants gave “Coming to LEAP helps me with 
my schoolwork” an average rating of 2.80 on a 3-point scale. 


Regional libraries—as opposed to neighborhood branch libraries—are well-poised to serve both 
children and parents seeking specific resources. For instance, the children’s section of one library 
visited occupied an entire floor and included a play area, computers, and book stacks. The library 
also referred participants to resources beyond the LEAP program and hosted a Literacy Lab for 
tutoring by a certified teacher. 


The central library, located near the Philadelphia museums, has a dedicated children’s section as well. 
Although the LEAP program at this location does not have a large core of regular participants, the 
program and library serve as a destination for special events or for participants seeking specific 
resources. This highlights an important distinction in that LEAP participants spend more time in 
smaller neighborhood libraries than they do in larger regional libraries. A librarian described this 
difference: 


I'd say that in the neighborhood libraries you get more of the kids who either go to school and 
live in the neighborhood, or go to school in the neighborhood and come to [the library] until 
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they can get picked up bya parent. In [a regional library] kids spend less time. They come here 
specifically to participate in the activities that we have. 


LEAP staff and participants report positive impacts on students’ 
self-confidence and self-reliance for learning 


LEAP staff aim to help children become more self-confident and self-reliant in their learning, and in 
their reading especially. Much of this is accomplished through the individual attention and support 
that LEAP participants receive during homework help: all staff reported supporting literacy as part of 
homework help, either regularly or on as as-needed basis. 


A majority of LEAP staff surveyed reported increases in indicators of participants’ confidence and 
self-reliance, including confidence in reading and reading motivation (89 percent and 75 percent 
reported some or substantial improvement in these areas, respectively). Further, participants gave 
generally high ratings on questions such as “When | try, | can get good grades in school” and “At 
LEAP, | learn how to be a good learner,” with an average rating for this domain of 2.82 on a 3-point 
scale (Exhibit 12). In general, this self-confidence among PreK-3 LEAP participants was evident across 
LEAP sites: the average rating by library ranged from 2.29 to 3 (mean of 2.79). 


Exhibit 12: Participant reports of self-confidence 
and self-reliance for learning 


Average rating across 
all questions (N=1,958) 2.82 


When | try, | can get good 
grades in school (N=312) 2.88 


When | try, | can finish 


my homework (N=323) 2.85 


| feel comfortable asking 
for help with my 2.84 
homework (N=321) 


At LEAP, | learn how to be 


a good reader (N=378) ol 


| can use what | learn at 


LEAP in school (N=311) 2.78 


| keep working on an 
assignment, even when 
it's hard (N=313) 


1 2 3 


Exhibit reads: PreK-3 LEAP participants gave “When | try, | can get good 
grades in school” an average rating of 2.88 on a 3-point scale. 
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LEAP sites foster this confidence by creating a positive culture around literacy and learning. For 
example, one LEAP program instructs students to use positive self-talk when approaching homework 
or other challenging tasks. The ASL noted that many participants are not motivated to do their 
homework because they don’t feel that they can do it perfectly. LEAP strives to help “students feel 
successful and capable,” according to this ASL. 


In some sites, the librarian plays a direct role in working with participants by bringing a level of 
technical skill in instruction and literacy that many LEAP staff have not yet developed. In one program, 
for instance, the children’s librarian worked directly with younger children in developing basic literacy 
skills, “...recognizing that they might be in a stage where they are still working on letter recognition. 
They're only in first grade but they are still already kind of behind in certain respects.” This focused 
work was amplified by the activities planned by the LEAP staff, including games, story cubes, and read- 
alouds. In another program, the librarian worked directly with participants to help them find 
appropriate books at their reading level or matched to their interest, while the ASL and TLAs focused 
on homework supervision and the activity planned for the day. 


Promoting Literacy Confidence in Judgement-Free Zone 


A librarian at a site where children expressed high levels of self-confidence in learning described a 
primary function of LEAP as engaging children in enjoyable reading-related activities that focus on 
“having a positive experience, that you won't be judged on.” The ASL at this site embraced this goal of 
creating a positive, judgement-free zone for learning: 


‘A lot of kids are anxious about getting every word and every sound correct immediately. If 
you don’t treat them like that's what they have to do, that’s a big part of doing informal 
literacy learning. I'm not going to make you feel shame, I'm going to help you sound it out 
if needed. If you tell me you can’t read well or don't like to read, | might offer an 
alternative truth: I’ve seen you read before, let’s do this together.” 


At this site, we observed some students struggling with assignments and seeking out support from the 
LEAP staff. For example, one child's school assignment was to analyze a fable. The ASL worked with him, 
checking for understanding through key prompts (“What is a character? Do you know what the setting 
is?”) and providing encouragement (“You did a good job defining a setting!”). 


At this same LEAP program, the ASL sought to create “a comfortable, self-initiated learning environment 
rather than a classroom environment... | work with kids on making choices and laying out their learning 
process so that they feel they are in charge.” For instance, he works with students to make a plan for 
completing their weekly homework packet or their reading log for the week or month, rather than 
enforcing completion during the program. “| try to communicate and work from a mindset that it’s not 
about homework completion but understanding what we do. We're building reading practice by reading 
over the passage and building critical problem-solving skills.” 
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CONCLUSIONS: THE ADDED 
VALUE OF LEAP WITHIN THE 
PHILADELPHIA OST SYSTEM 


As a drop-in, library-based program, LEAP offers a unique contribution to the OST system in 
Philadelphia. 


First, the drop-in nature of the program makes the services accessible to families who do not want to 
commit to a daily afterschool program but need to take advantage of the resources and assistance 
provided on occasion. In addition to providing the flexibility for parents, the drop-in nature of the 
program appeals to children who may not want or need the structure of being enrolled in a daily 
afterschool program. In this way, LEAP complements other opportunities available through the city 
OST system; together these elements create a comprehensive menu of services that meets the needs 
of a wider variety of families and interests, as summarized by one ASL: 


It's a free and voluntary program that you can attend as much or as little without any penalty 
whatsoever. You don't have to work your life around it. There's no mandatory activity that you 
must do. It’s all up to the child and whoever brings them. 


| love it because it's really independent, it's drop-in.... Nobody's forcing you to come every day, 
so it's almost like you come here because you like it, and you know it's a safe place to be. It's a 
fun environment, and that there's resources. You can use the computer and it doesn't cost 
additional money... It's a place that they count on to get assistance in homework help for 
anything that they might need that they don't understand during the day. 


Second, LEAP provides an opportunity for whole-family learning, creating a space in which children, 
parents and other caregivers can learn together, particularly as participants ages 7 and under are 
required to be accompanied to the program. One librarian described LEAP as “the new community 
center” to which both parents and children can come for free assistance and support, aligned with the 
mission of the library. The connection to the library amplifies the resources available to participating 
children and families: 


| think it's good that [the OST program] is part of the Free Library of Philadelphia. So it's 
embedded in that network, where you have all these resources, including right here in our 
neighborhood library, and you can connect to people with the larger library system. So if people 
needed certain resources you can refer them. [It’s] having a larger pool of people, of resources 
and people at your fingertips to connect the kids. 
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Third, LEAP distinguishes itself from other city-funded OST programs through its library setting. 
Although LEAP does not describe itself as a literacy instruction program, it connects children to 
books and engaging literacy experiences, and encourages reading behaviors and confidence. LEAP 
and the library offer valuable resources for literacy development that connect to the individual 
learning needs and interests of participants. One ASL summed this up by simply stating: “We're a 
library. We have books, we have a computer, we have anything that could grab a student's interest.” 


Recommendations 


For LEAP staff 


Our evaluation identified three practice recommendations for LEAP staff to strengthen the overall 
implementation and impact of LEAP: 


1. 


Build on the strengths of the library as a community institution for both children and 
families to support learning and ensure success in navigating the school system. LEAP 
staff can help both children and the caregivers who accompany them develop skills by 
modeling homework help and encouraging parents in enhancing strategies for learning 
support. As one ASL noted, “It's multiplying the impact if mom, dad, or grandma is learning 
alongside the child.” Research confirms the value of engaging parents in supporting student 
reading efforts (Osterhaus, n.d.; Ishimaru & Lott, 2014). 


Be intentional about literacy, and about embedding literacy enrichment activities. |n 
some sites, we observed LEAP staff asking children probing questions to increase reading 
comprehension during homework assistance. During arts and STEM activities, staff reinforced 
literacy through clear connections to a book or a story. Indeed, LEAP staff must balance their 
goals of not only offering youth-driven activities but also intentionally engaging young 
people in literacy and scaffolding literacy skills. Implementing literacy-focused activities can 
be a challenge for LEAP staff: “I personally see reading as much more of an independent 
activity, and the kids are always interacting. | think maybe getting more ideas and training 
around how to do group activities with literacy [would be helpful], not just sitting and 
reading a book to them.” 


Leverage library resources and relationships with LEAP participants to encourage 
reading behaviors. Some LEAP staff reported that they struggle to engage reluctant readers. 
“It would be nice if some of the kids were reading more,” said one staff member. “Sometimes 
| will suggest a book to read. But some kids don’t even want to try.” Other staff have 
developed solutions for this common challenge—for instance, finding books that are relevant 
to students’ interests in videogames or that connect to a popular movie (such as the 
Minecraft or Fortnite series of books). Creating an online resource or community of practice 
for LEAP staff to share strategies and ideas could help them to address and resolve this 
common challenge. 
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For the Free Library of Philadelphia 


The Free Library can play an essential role in enabling LEAP staff to implement the practice 
recommendations above, by providing training specifically focused on literacy instruction. 


The Youth Services and Programs division of the Free Library already offers considerable training and 
resources to LEAP staff through its hiring and onboarding processes, monthly curriculum trainings, 
and periodic all-staff meetings. This training platform can be leveraged to more explicitly build the 
capacity of LEAP staff to provide literacy support. The Free Library can also capitalize on the 
experience and expertise of LEAP staff and librarians who have locally developed promising 
strategies for intentionally encouraging and modeling literacy skills and behaviors, through cross-site 
peer-learning and mentoring opportunities. 


Both LEAP staff and librarians are eager to develop their capacity to support literacy development, as 
summarized by one librarian: “I didn’t study education and so | think there are still ways for me to 
understand the best methods and best practices for instructing literacy and knowing ... how to work 
with different learning abilities.” 


With additional training and opportunities to share best practices and resources, LEAP is poised to 
deepen its impact on early literacy and to capitalize on its role as a welcoming, accessible resource 
for both occasional and regular participants and their families. 


For the City of Philadelphia 


As the City of Philadelphia continues to invest in and support its 2018-2026 OST strategic plan, 
lessons learned from the LEAP evaluation point to refine metrics for success for LEAP, given the 
distinct role of LEAP within the comprehensive citywide OST system. The patterns of utilization for 
LEAP in particular—with both a core group of regular participants and a significant proportion of 
occasional participants— is reflective of LEAP’s goals and expectations as a drop-in program. 
Therefore, we recommend that metrics for LEAP’s success and contribution to the Philadelphia’s OST 
system reflect the value of serving families and children in a range of ways; assumptions underlying 
typical OST dosage measures are inconsistent with LEAP design. 
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APPENDIX: 
RESEARCH QUESTIONS, 
METHODS, AND DATA SOURCES 


Research Questions 


Consistent with the City of Philadelphia’s priority on supporting the Read by 4‘ campaign as part of 
its OST strategic plan, the evaluation focused on the experiences of participants in grades PresK-3 
and addressed the following questions about the implementation and impact of LEAP: 


m What do patterns of participation in LEAP reveal about program operation and potential 
impact? 


m What strategies do LEAP staff implement to promote foundational literacy and social- 
emotional skills? 


In what ways do LEAP staff leverage and adapt the resources offered by the Free Library of 
Philadelphia? 


m What are the effects of LEAP on the development of skills and attitudes that will help young 
people succeed in school and beyond? 


m= What recommendations for improvement emerge from this analysis of implementation and 
outcomes? 


In our analyses, we also explored whether the experience and benefit of LEAP varied based on the 
level of youth engagement. 


Staff Survey 


In February 2019, PSA surveyed LEAP staff—ASLs, MMs, and librarians—at the 51 Free Library 
branches with active LEAP programs, using an online survey. We received completed surveys from at 
least one staff member at every library, with an overall staff response rate of 78 percent. The survey 
asked staff about their background, their role and responsibilities for LEAP programming at their 
library, the reasons they believe youth begin and then continue attending LEAP, the types of literacy- 
related activities their LEAP program provides, and their perceptions of the impacts of the program 
on youth who attend. 
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Participant Survey 


PSA surveyed participants over a 10-week period between February and April 2019. The survey and 
administration were designed to minimize the impact on regular program operations while reaching 
as many participants as possible. We considered two important factors when designing the survey. 
First, we paid careful attention to participants’ privacy and confidentiality given their age and the 
structure of the LEAP program. LEAP has no formal sign-up process and no assumption that 
guardians will accompany children to the library to give consent for the collection of any personally 
identifying information. Also, libraries value patrons’ privacy, and we needed to be sensitive to the 
concerns of library and LEAP staff. Second, LEAP is a drop-in program, with no expectation that a 
participant will attend multiple days in the same week or even the same month. This complicates 
efforts to track responses from individual participants over time. 


We worked in consultation with an advisory group of ASLs to design a survey to administer that 
addressed these limitations. We developed 31 survey questions aligned with the key LEAP domains: 
safe and supportive OST environments; self-reliance and self-confidence for learning; love of reading; 
collaboration and teamwork skills; strong relationships; and knowledge of library resources. Twenty- 
three of the 33 questions were measured on a 3-point “smiley face Likert scale,” an appropriate 
measurement for children in our focus age group (PreK-3) (Davies & Brember, 1994). An additional 
question asked youth to circle the time they spent reading each day (less than 30 minutes, 30 
minutes to an hour; more than an hour). The remaining nine questions used a “yes/no” format. The 
3-point questions were then distributed to libraries over a period of weeks, with two to three 
questions per week, to ensure a distribution of questions in the same domain across different weeks; 
all of the yes/no questions were asked in the same week. Every week, we asked two additional 
questions—"What grade are you in?” and “Is this your first time at LEAP?” 


These “Questions of the Week” were printed on cards, sorted and packaged by week, and then sent 
to the libraries with small incentives (e.g., a small toy, an eraser, or pencil) and guidelines for 
administering the questions and returning the completed survey cards to PSA. We received 8,499 
completed survey cards (3,826 from participants in PreK-3) from 48 of the 51 libraries to which we 
sent survey administration materials, which amounted to approximately 70 percent of the cards sent 
to libraries. The survey responses were well-distributed across weeks, and the number of survey 
cards received from each library from each week roughly matched what was expected based on 
participant numbers extracted from the Philadelphia Citywide OST database (described below). 


Participation Data 


For this report, PSA requested extracts of data from the Philadelphia Citywide OST database hosted 
by Cityspan, for program years 2015-16, 2016-17, 2017-18, and 2018-19 (September through June). 
These extracts contained information on participant characteristics, including grade level, age, 
gender, race, ethnicity, primary language spoken at home, home zip code, and school attended. Data 
also included a record for each date on which a participant visited a LEAP program and the library 
attended. Although there is no formal enrollment process, LEAP program staff ideally register every 
child attending LEAP in the Cityspan database, recording individual characteristics (e.g., age, gender, 
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race, school), and subsequently recording every day that child attended LEAP at their site. However, 
based on discussions with staff from the Free Library and confirmed by our assessment of the data, 
we know that from one library to the next, the decision about when to enter a participant varies: 
some library staff enter a child on the first visit, while others wait to see if the child becomes a 
recurring participant. In our analyses, we note the percent of participants for whom we are missing 
data. Still, participation data must be viewed with some caution, since we know that data for some 
participants are missing, but we do not know to what extent or whether the absence of data is 
related to participant or library characteristics. 


Site Visits and Interviews 


In February and March 2019, we conducted one-day visits to 10 LEAP sites. The sites were selected in 
conjunction with Free Library staff based on several criteria: 


m High proportions of participants served in preK-3 

m Planned implementation of a range of approaches to supporting learning 
m@ Locations in a range of community contexts 

m™ Varied patterns of participation in weekday and Saturday program 


During these site visits, the PSA evaluation team interviewed LEAP ASLs and librarians and observed 
LEAP programming. 
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